So far every pathway analysis method we’ve covered relies on some information about groups of samples in our data. For over-representation analysis (ORA), we created gene sets from two different two group comparisons. In the Gene Set Enrichment Analysis (GSEA) example, we used statistics from a differential gene expression (DGE) analysis where we compared MYCN amplified cell lines to non-amplified cell lines; we needed that amplification status information.

What if we’re less sure about groups in our data or we want to analyze our data in a more unsupervised manner?

In this notebook we will cover a method called Gene Set Variation Analysis (GSVA) (Hänzelmann et al. 2013) that allows us to calculate gene set or pathway scores on a per-sample basis.

We like this quote from the GSVA paper (Hänzelmann et al. 2013) to set the stage:

While [gene set enrichment] methods are generally regarded as end points of a bioinformatic analysis, GSVA constitutes a starting point to build pathway-centric models of biology.

Rather than contextualizing some results you already have from another analysis like DGE, GSVA is designed to provide an estimate of pathway variation for each of the samples in an experiment. Note that these scores will depend on the samples included in the dataset when you run GSVA; if you added more samples and reran GSVA, you would expect the scores to change.

Set up

Libraries

# Pipes
library(magrittr)
# Gene Set Variation Analysis
library(GSVA)

Directories and files

Directories

# We have some medulloblastoma data from the OpenPBTA project that we've 
# prepared ahead of time
input_dir <- file.path("data", "open-pbta")

# Create a directory specifically for the results using this dataset
output_dir <- file.path("results", "open-pbta") 
if (!dir.exists(output_dir)) {
  dir.create(output_dir, recursive = TRUE)
}

Input

We have VST transformed RNA-seq data, annotated with gene symbols, that has been collapsed such that there are no duplicated gene identifiers (see setup).

rnaseq_file <- file.path(input_dir, "medulloblastoma_vst_collapsed.tsv")

Output

gsva_results_file <- file.path(output_dir, "medulloblastoma_gsva_results.tsv")

Gene sets

The function that we will use to run GSVA wants the gene sets to be in a list, rather than a tidy data frame that we used with clusterProfiler (although it does accept other formats).

We’re going to take this opportunity to introduce a different format that gene sets are often distributed in called GMT (Gene Matrix Transposed).

We’re going to read in the Hallmark collection file directly from MSigDB, rather than using msigdbr like we did in earlier notebooks.

The RNA-seq data uses gene symbols, so we need gene sets that use gene symbols, too.

# R can often read in data from a URL
hallmarks_url <- "https://data.broadinstitute.org/gsea-msigdb/msigdb/release/7.1/h.all.v7.1.symbols.gmt"

# QuSAGE is another pathway analysis method, the qusage package has a function
# for reading GMT files and turning them into a list
hallmarks_list <- qusage::read.gmt(hallmarks_url)

What does this list look like?

head(hallmarks_list)

RNA-seq data

We have VST transformed RNA-seq data, which is on a log2-like scale. These data are from the Open Pediatric Brain Tumor Atlas (OpenPBTA) OpenPBTA is a collaborative project organized by the CCDL and the Center for Data-Driven Discovery in Biomedicine (D3b) at the Children’s Hospital of Philadelphia conducted openly on GitHub.

You can read more about the project here.

We’re only working with the medulloblastoma samples in this example.

rnaseq_df <- readr::read_tsv(rnaseq_file)

── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
cols(
  .default = col_double(),
  gene_symbol = col_character()
)
ℹ Use `spec()` for the full column specifications.
# What does the RNA-seq data frame look like?
rnaseq_df[1:5, 1:5]

For GSVA, we need a matrix.

rnaseq_mat <- rnaseq_df %>%
  tibble::column_to_rownames("gene_symbol") %>%
  as.matrix()

Note: If we had duplicate gene symbols here, we couldn’t set them as rownames.

GSVA

Figure 1 from Hänzelmann et al. (2013).

You may notice that GSVA has some commonalities with GSEA. Rather than ranking genes based on some statistic we selected ahead of time, GSVA fits a model and ranks genes based on their expression level relative to the sample distribution. This is a way of asking if a gene i is highly or lowly expressed in a sample j in the context of this experiment and ranking accordingly (Hänzelmann et al. 2013). The pathway-level score calculated is a way of asking how genes within a gene set vary as compared to genes that are outside of that gene set (Malhotra. 2018). (This is sometimes called a competitive test in gene set enrichment literature.) The intuition here is that we will get pathway-level scores for each sample that indicate if genes in a pathway vary concordantly in one direction (overexpressed or underexpressed relative to the overall population) (Hänzelmann et al. 2013).

The output is a gene set by sample matrix of GSVA scores.

Perform GSVA

gsva_results <- gsva(rnaseq_mat, 
                     hallmarks_list,
                     method = "gsva",
                     # Appropriate for our transformed data on log2-like scale
                     kcdf = "Gaussian",  
                     # Minimum gene set size
                     min.sz = 15, 
                     # Maximum gene set size
                     max.sz = 500,
                     # Compute Gaussian-distributed scores
                     mx.diff = TRUE)
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 50 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                                                     
  |                                                                                                                               |   0%
  |                                                                                                                                     
  |===                                                                                                                            |   2%
  |                                                                                                                                     
  |=====                                                                                                                          |   4%
  |                                                                                                                                     
  |========                                                                                                                       |   6%
  |                                                                                                                                     
  |==========                                                                                                                     |   8%
  |                                                                                                                                     
  |=============                                                                                                                  |  10%
  |                                                                                                                                     
  |===============                                                                                                                |  12%
  |                                                                                                                                     
  |==================                                                                                                             |  14%
  |                                                                                                                                     
  |====================                                                                                                           |  16%
  |                                                                                                                                     
  |=======================                                                                                                        |  18%
  |                                                                                                                                     
  |=========================                                                                                                      |  20%
  |                                                                                                                                     
  |============================                                                                                                   |  22%
  |                                                                                                                                     
  |==============================                                                                                                 |  24%
  |                                                                                                                                     
  |=================================                                                                                              |  26%
  |                                                                                                                                     
  |====================================                                                                                           |  28%
  |                                                                                                                                     
  |======================================                                                                                         |  30%
  |                                                                                                                                     
  |=========================================                                                                                      |  32%
  |                                                                                                                                     
  |===========================================                                                                                    |  34%
  |                                                                                                                                     
  |==============================================                                                                                 |  36%
  |                                                                                                                                     
  |================================================                                                                               |  38%
  |                                                                                                                                     
  |===================================================                                                                            |  40%
  |                                                                                                                                     
  |=====================================================                                                                          |  42%
  |                                                                                                                                     
  |========================================================                                                                       |  44%
  |                                                                                                                                     
  |==========================================================                                                                     |  46%
  |                                                                                                                                     
  |=============================================================                                                                  |  48%
  |                                                                                                                                     
  |================================================================                                                               |  50%
  |                                                                                                                                     
  |==================================================================                                                             |  52%
  |                                                                                                                                     
  |=====================================================================                                                          |  54%
  |                                                                                                                                     
  |=======================================================================                                                        |  56%
  |                                                                                                                                     
  |==========================================================================                                                     |  58%
  |                                                                                                                                     
  |============================================================================                                                   |  60%
  |                                                                                                                                     
  |===============================================================================                                                |  62%
  |                                                                                                                                     
  |=================================================================================                                              |  64%
  |                                                                                                                                     
  |====================================================================================                                           |  66%
  |                                                                                                                                     
  |======================================================================================                                         |  68%
  |                                                                                                                                     
  |=========================================================================================                                      |  70%
  |                                                                                                                                     
  |===========================================================================================                                    |  72%
  |                                                                                                                                     
  |==============================================================================================                                 |  74%
  |                                                                                                                                     
  |=================================================================================================                              |  76%
  |                                                                                                                                     
  |===================================================================================================                            |  78%
  |                                                                                                                                     
  |======================================================================================================                         |  80%
  |                                                                                                                                     
  |========================================================================================================                       |  82%
  |                                                                                                                                     
  |===========================================================================================================                    |  84%
  |                                                                                                                                     
  |=============================================================================================================                  |  86%
  |                                                                                                                                     
  |================================================================================================================               |  88%
  |                                                                                                                                     
  |==================================================================================================================             |  90%
  |                                                                                                                                     
  |=====================================================================================================================          |  92%
  |                                                                                                                                     
  |=======================================================================================================================        |  94%
  |                                                                                                                                     
  |==========================================================================================================================     |  96%
  |                                                                                                                                     
  |============================================================================================================================   |  98%
  |                                                                                                                                     
  |===============================================================================================================================| 100%

Note: the gsva() documentation says we can use kcdf = "Gaussian" if we had RNA-seq log-CPMs, log-RPKMs or log-TPMs, but we would use kcdf = "Poisson" on integer counts.

# Let's explore what the output of gsva() looks like
gsva_results[1:5, 1:5]
                                    BS_09Z7TC35 BS_1AYRM596 BS_1BWP5MCT BS_1QXEC43H BS_1TWCV047
HALLMARK_TNFA_SIGNALING_VIA_NFKB    -0.44911439   0.5208667  -0.5609193  0.32300630  -0.1468081
HALLMARK_HYPOXIA                    -0.38297104   0.2436910  -0.5058759  0.36247083  -0.2971559
HALLMARK_CHOLESTEROL_HOMEOSTASIS    -0.26534735   0.1054224  -0.5180933  0.32418657  -0.4561386
HALLMARK_MITOTIC_SPINDLE             0.12727006   0.2339489  -0.4338076 -0.07023068  -0.1861483
HALLMARK_WNT_BETA_CATENIN_SIGNALING -0.09287646   0.1602124  -0.4427594  0.41372523  -0.1152437

A note on gene set size

Often the scores of gene set enrichment methods are not comparable between gene sets of different sizes. Let’s do an experiment using randomly generated gene sets to explore this idea a bit more.

We need to get a collection of all possible genes we will sample from to create random gene sets. Because we’re doing some random sampling, we need to set a seed for this to be reproducible.

# Use all the gene symbols in the dataset as the pool of possible genes
all_genes <- rownames(rnaseq_mat)

# Set a seed for reproducibility
set.seed(2020)

Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. We’ll use the same minimum and maximum values for our random gene sets and some values in between.

# Make a list of integers that indicate the random gene set sizes
gene_set_size <- list(15, 25, 50, 100, 250, 500)

For each gene set size, we will generate 100 random gene sets

# Set number of replicates
nreps <-  100
# Generate 100 random gene sets of each size
random_gene_sets <- 
  purrr::map(
    rep(gene_set_size, nreps),  # Repeat gene sizes so we run `nreps` times 
    # Sample the vector of all genes, choosing the number of items specified
    # in the element of gene set size
    ~ base::sample(x = all_genes,
                   size = .x)
  )

The Hallmarks list we used earlier stored the gene set names as the name of the list, so let’s add names to our random gene sets that indicate what size they are and so gsva() doesn’t get upset.

# We will include the size of the gene set in the gene set name 
# Start by taking the length of each pathway and appending "pathway_" to that
# number
lengths_vector <- purrr::map(random_gene_sets, ~ length(.x)) %>%
  purrr::map(~ paste0("pathway_", .x)) %>%
  # Return a vector
  purrr::flatten_chr()

# Add the names in lengths_vector to the list
random_gene_sets <- random_gene_sets %>% 
  # make.names() appends a "version" if something is not unique
  purrr::set_names(nm = make.names(lengths_vector, unique = TRUE))

Run GSVA on our dataset with the same parameters as before, but now with random gene sets.

random_gsva_results <- gsva(rnaseq_mat, 
                            random_gene_sets,
                            method = "gsva",
                            # Appropriate for our transformed data on 
                            # log2-like scale
                            kcdf = "Gaussian",  
                            # Minimum gene set size
                            min.sz = 15, 
                            # Maximum gene set size
                            max.sz = 500,
                            # Compute Gaussian-distributed scores
                            mx.diff = TRUE)                            
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 579 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                                                     
  |                                                                                                                               |   0%
  |                                                                                                                                     
  |=                                                                                                                              |   1%
  |                                                                                                                                     
  |==                                                                                                                             |   1%
  |                                                                                                                                     
  |==                                                                                                                             |   2%
  |                                                                                                                                     
  |===                                                                                                                            |   2%
  |                                                                                                                                     
  |===                                                                                                                            |   3%
  |                                                                                                                                     
  |====                                                                                                                           |   3%
  |                                                                                                                                     
  |=====                                                                                                                          |   4%
  |                                                                                                                                     
  |======                                                                                                                         |   4%
  |                                                                                                                                     
  |======                                                                                                                         |   5%
  |                                                                                                                                     
  |=======                                                                                                                        |   5%
  |                                                                                                                                     
  |=======                                                                                                                        |   6%
  |                                                                                                                                     
  |========                                                                                                                       |   6%
  |                                                                                                                                     
  |========                                                                                                                       |   7%
  |                                                                                                                                     
  |=========                                                                                                                      |   7%
  |                                                                                                                                     
  |==========                                                                                                                     |   8%
  |                                                                                                                                     
  |===========                                                                                                                    |   8%
  |                                                                                                                                     
  |===========                                                                                                                    |   9%
  |                                                                                                                                     
  |============                                                                                                                   |   9%
  |                                                                                                                                     
  |============                                                                                                                   |  10%
  |                                                                                                                                     
  |=============                                                                                                                  |  10%
  |                                                                                                                                     
  |=============                                                                                                                  |  11%
  |                                                                                                                                     
  |==============                                                                                                                 |  11%
  |                                                                                                                                     
  |===============                                                                                                                |  12%
  |                                                                                                                                     
  |================                                                                                                               |  12%
  |                                                                                                                                     
  |================                                                                                                               |  13%
  |                                                                                                                                     
  |=================                                                                                                              |  13%
  |                                                                                                                                     
  |=================                                                                                                              |  14%
  |                                                                                                                                     
  |==================                                                                                                             |  14%
  |                                                                                                                                     
  |==================                                                                                                             |  15%
  |                                                                                                                                     
  |===================                                                                                                            |  15%
  |                                                                                                                                     
  |====================                                                                                                           |  15%
  |                                                                                                                                     
  |====================                                                                                                           |  16%
  |                                                                                                                                     
  |=====================                                                                                                          |  16%
  |                                                                                                                                     
  |=====================                                                                                                          |  17%
  |                                                                                                                                     
  |======================                                                                                                         |  17%
  |                                                                                                                                     
  |======================                                                                                                         |  18%
  |                                                                                                                                     
  |=======================                                                                                                        |  18%
  |                                                                                                                                     
  |========================                                                                                                       |  19%
  |                                                                                                                                     
  |=========================                                                                                                      |  19%
  |                                                                                                                                     
  |=========================                                                                                                      |  20%
  |                                                                                                                                     
  |==========================                                                                                                     |  20%
  |                                                                                                                                     
  |==========================                                                                                                     |  21%
  |                                                                                                                                     
  |===========================                                                                                                    |  21%
  |                                                                                                                                     
  |===========================                                                                                                    |  22%
  |                                                                                                                                     
  |============================                                                                                                   |  22%
  |                                                                                                                                     
  |=============================                                                                                                  |  22%
  |                                                                                                                                     
  |=============================                                                                                                  |  23%
  |                                                                                                                                     
  |==============================                                                                                                 |  23%
  |                                                                                                                                     
  |==============================                                                                                                 |  24%
  |                                                                                                                                     
  |===============================                                                                                                |  24%
  |                                                                                                                                     
  |===============================                                                                                                |  25%
  |                                                                                                                                     
  |================================                                                                                               |  25%
  |                                                                                                                                     
  |================================                                                                                               |  26%
  |                                                                                                                                     
  |=================================                                                                                              |  26%
  |                                                                                                                                     
  |==================================                                                                                             |  26%
  |                                                                                                                                     
  |==================================                                                                                             |  27%
  |                                                                                                                                     
  |===================================                                                                                            |  27%
  |                                                                                                                                     
  |===================================                                                                                            |  28%
  |                                                                                                                                     
  |====================================                                                                                           |  28%
  |                                                                                                                                     
  |====================================                                                                                           |  29%
  |                                                                                                                                     
  |=====================================                                                                                          |  29%
  |                                                                                                                                     
  |======================================                                                                                         |  30%
  |                                                                                                                                     
  |=======================================                                                                                        |  30%
  |                                                                                                                                     
  |=======================================                                                                                        |  31%
  |                                                                                                                                     
  |========================================                                                                                       |  31%
  |                                                                                                                                     
  |========================================                                                                                       |  32%
  |                                                                                                                                     
  |=========================================                                                                                      |  32%
  |                                                                                                                                     
  |=========================================                                                                                      |  33%
  |                                                                                                                                     
  |==========================================                                                                                     |  33%
  |                                                                                                                                     
  |===========================================                                                                                    |  34%
  |                                                                                                                                     
  |============================================                                                                                   |  34%
  |                                                                                                                                     
  |============================================                                                                                   |  35%
  |                                                                                                                                     
  |=============================================                                                                                  |  35%
  |                                                                                                                                     
  |=============================================                                                                                  |  36%
  |                                                                                                                                     
  |==============================================                                                                                 |  36%
  |                                                                                                                                     
  |===============================================                                                                                |  37%
  |                                                                                                                                     
  |================================================                                                                               |  37%
  |                                                                                                                                     
  |================================================                                                                               |  38%
  |                                                                                                                                     
  |=================================================                                                                              |  38%
  |                                                                                                                                     
  |=================================================                                                                              |  39%
  |                                                                                                                                     
  |==================================================                                                                             |  39%
  |                                                                                                                                     
  |==================================================                                                                             |  40%
  |                                                                                                                                     
  |===================================================                                                                            |  40%
  |                                                                                                                                     
  |====================================================                                                                           |  41%
  |                                                                                                                                     
  |=====================================================                                                                          |  41%
  |                                                                                                                                     
  |=====================================================                                                                          |  42%
  |                                                                                                                                     
  |======================================================                                                                         |  42%
  |                                                                                                                                     
  |======================================================                                                                         |  43%
  |                                                                                                                                     
  |=======================================================                                                                        |  43%
  |                                                                                                                                     
  |=======================================================                                                                        |  44%
  |                                                                                                                                     
  |========================================================                                                                       |  44%
  |                                                                                                                                     
  |=========================================================                                                                      |  45%
  |                                                                                                                                     
  |==========================================================                                                                     |  45%
  |                                                                                                                                     
  |==========================================================                                                                     |  46%
  |                                                                                                                                     
  |===========================================================                                                                    |  46%
  |                                                                                                                                     
  |===========================================================                                                                    |  47%
  |                                                                                                                                     
  |============================================================                                                                   |  47%
  |                                                                                                                                     
  |=============================================================                                                                  |  48%
  |                                                                                                                                     
  |==============================================================                                                                 |  49%
  |                                                                                                                                     
  |===============================================================                                                                |  49%
  |                                                                                                                                     
  |===============================================================                                                                |  50%
  |                                                                                                                                     
  |================================================================                                                               |  50%
  |                                                                                                                                     
  |================================================================                                                               |  51%
  |                                                                                                                                     
  |=================================================================                                                              |  51%
  |                                                                                                                                     
  |==================================================================                                                             |  52%
  |                                                                                                                                     
  |===================================================================                                                            |  53%
  |                                                                                                                                     
  |====================================================================                                                           |  53%
  |                                                                                                                                     
  |====================================================================                                                           |  54%
  |                                                                                                                                     
  |=====================================================================                                                          |  54%
  |                                                                                                                                     
  |=====================================================================                                                          |  55%
  |                                                                                                                                     
  |======================================================================                                                         |  55%
  |                                                                                                                                     
  |=======================================================================                                                        |  56%
  |                                                                                                                                     
  |========================================================================                                                       |  56%
  |                                                                                                                                     
  |========================================================================                                                       |  57%
  |                                                                                                                                     
  |=========================================================================                                                      |  57%
  |                                                                                                                                     
  |=========================================================================                                                      |  58%
  |                                                                                                                                     
  |==========================================================================                                                     |  58%
  |                                                                                                                                     
  |==========================================================================                                                     |  59%
  |                                                                                                                                     
  |===========================================================================                                                    |  59%
  |                                                                                                                                     
  |============================================================================                                                   |  60%
  |                                                                                                                                     
  |=============================================================================                                                  |  60%
  |                                                                                                                                     
  |=============================================================================                                                  |  61%
  |                                                                                                                                     
  |==============================================================================                                                 |  61%
  |                                                                                                                                     
  |==============================================================================                                                 |  62%
  |                                                                                                                                     
  |===============================================================================                                                |  62%
  |                                                                                                                                     
  |===============================================================================                                                |  63%
  |                                                                                                                                     
  |================================================================================                                               |  63%
  |                                                                                                                                     
  |=================================================================================                                              |  64%
  |                                                                                                                                     
  |==================================================================================                                             |  64%
  |                                                                                                                                     
  |==================================================================================                                             |  65%
  |                                                                                                                                     
  |===================================================================================                                            |  65%
  |                                                                                                                                     
  |===================================================================================                                            |  66%
  |                                                                                                                                     
  |====================================================================================                                           |  66%
  |                                                                                                                                     
  |=====================================================================================                                          |  67%
  |                                                                                                                                     
  |======================================================================================                                         |  67%
  |                                                                                                                                     
  |======================================================================================                                         |  68%
  |                                                                                                                                     
  |=======================================================================================                                        |  68%
  |                                                                                                                                     
  |=======================================================================================                                        |  69%
  |                                                                                                                                     
  |========================================================================================                                       |  69%
  |                                                                                                                                     
  |========================================================================================                                       |  70%
  |                                                                                                                                     
  |=========================================================================================                                      |  70%
  |                                                                                                                                     
  |==========================================================================================                                     |  71%
  |                                                                                                                                     
  |===========================================================================================                                    |  71%
  |                                                                                                                                     
  |===========================================================================================                                    |  72%
  |                                                                                                                                     
  |============================================================================================                                   |  72%
  |                                                                                                                                     
  |============================================================================================                                   |  73%
  |                                                                                                                                     
  |=============================================================================================                                  |  73%
  |                                                                                                                                     
  |=============================================================================================                                  |  74%
  |                                                                                                                                     
  |==============================================================================================                                 |  74%
  |                                                                                                                                     
  |===============================================================================================                                |  74%
  |                                                                                                                                     
  |===============================================================================================                                |  75%
  |                                                                                                                                     
  |================================================================================================                               |  75%
  |                                                                                                                                     
  |================================================================================================                               |  76%
  |                                                                                                                                     
  |=================================================================================================                              |  76%
  |                                                                                                                                     
  |=================================================================================================                              |  77%
  |                                                                                                                                     
  |==================================================================================================                             |  77%
  |                                                                                                                                     
  |==================================================================================================                             |  78%
  |                                                                                                                                     
  |===================================================================================================                            |  78%
  |                                                                                                                                     
  |====================================================================================================                           |  78%
  |                                                                                                                                     
  |====================================================================================================                           |  79%
  |                                                                                                                                     
  |=====================================================================================================                          |  79%
  |                                                                                                                                     
  |=====================================================================================================                          |  80%
  |                                                                                                                                     
  |======================================================================================================                         |  80%
  |                                                                                                                                     
  |======================================================================================================                         |  81%
  |                                                                                                                                     
  |=======================================================================================================                        |  81%
  |                                                                                                                                     
  |========================================================================================================                       |  82%
  |                                                                                                                                     
  |=========================================================================================================                      |  82%
  |                                                                                                                                     
  |=========================================================================================================                      |  83%
  |                                                                                                                                     
  |==========================================================================================================                     |  83%
  |                                                                                                                                     
  |==========================================================================================================                     |  84%
  |                                                                                                                                     
  |===========================================================================================================                    |  84%
  |                                                                                                                                     
  |===========================================================================================================                    |  85%
  |                                                                                                                                     
  |============================================================================================================                   |  85%
  |                                                                                                                                     
  |=============================================================================================================                  |  85%
  |                                                                                                                                     
  |=============================================================================================================                  |  86%
  |                                                                                                                                     
  |==============================================================================================================                 |  86%
  |                                                                                                                                     
  |==============================================================================================================                 |  87%
  |                                                                                                                                     
  |===============================================================================================================                |  87%
  |                                                                                                                                     
  |===============================================================================================================                |  88%
  |                                                                                                                                     
  |================================================================================================================               |  88%
  |                                                                                                                                     
  |=================================================================================================================              |  89%
  |                                                                                                                                     
  |==================================================================================================================             |  89%
  |                                                                                                                                     
  |==================================================================================================================             |  90%
  |                                                                                                                                     
  |===================================================================================================================            |  90%
  |                                                                                                                                     
  |===================================================================================================================            |  91%
  |                                                                                                                                     
  |====================================================================================================================           |  91%
  |                                                                                                                                     
  |====================================================================================================================           |  92%
  |                                                                                                                                     
  |=====================================================================================================================          |  92%
  |                                                                                                                                     
  |======================================================================================================================         |  93%
  |                                                                                                                                     
  |=======================================================================================================================        |  93%
  |                                                                                                                                     
  |=======================================================================================================================        |  94%
  |                                                                                                                                     
  |========================================================================================================================       |  94%
  |                                                                                                                                     
  |========================================================================================================================       |  95%
  |                                                                                                                                     
  |=========================================================================================================================      |  95%
  |                                                                                                                                     
  |=========================================================================================================================      |  96%
  |                                                                                                                                     
  |==========================================================================================================================     |  96%
  |                                                                                                                                     
  |===========================================================================================================================    |  97%
  |                                                                                                                                     
  |============================================================================================================================   |  97%
  |                                                                                                                                     
  |============================================================================================================================   |  98%
  |                                                                                                                                     
  |=============================================================================================================================  |  98%
  |                                                                                                                                     
  |=============================================================================================================================  |  99%
  |                                                                                                                                     
  |============================================================================================================================== |  99%
  |                                                                                                                                     
  |===============================================================================================================================| 100%

Now let’s make a plot to look at the distribution of scores from random gene sets. First we need to get this data in an appropriate format for ggplot2.

# The random results are a matrix
random_long_df <- random_gsva_results %>%
  data.frame() %>%
  # Gene set names are rownames
  tibble::rownames_to_column("gene_set") %>%
  # Get into long format
  tidyr::pivot_longer(cols = -gene_set, 
                      names_to = "Kids_First_Biospecimen_ID",
                      values_to = "gsva_score") %>%
  # Remove the .version added by make.names()
  dplyr::mutate(gene_set = stringr::str_remove(gene_set, "\\..*")) %>%
  # Add a column that keeps track of the gene set size
  dplyr::mutate(gene_set_size = stringr::word(gene_set, 2, sep = "_")) %>%
  # We want to plot smallest no. genes -> largest no. genes
  dplyr::mutate(gene_set_size = factor(gene_set_size, 
                                       levels = c(15, 25, 50, 100, 250, 500)))

Let’s make a violin plot so we can look at the distribution of scores by gene set size.

# Violin plot comparing GSVA scores of different random gene set sizes
random_long_df %>%
  ggplot2::ggplot(ggplot2::aes(x = gene_set_size, 
                               y = gsva_score)) +
  # Make a violin plot that's a pretty blue!
  ggplot2::geom_violin(fill = "#99CCFF", alpha = 0.5) +
  # Add a point with the mean value
  ggplot2::stat_summary(
    geom = "point",
    fun = "mean",
    # Change the aesthetics of the points
    size = 3,
    color = "#0066CC",
    shape = 18
  ) +
  # Flip the axes
  ggplot2::coord_flip() +
  ggplot2::labs(title = "Random Gene Set GSVA Scores",
                x = "gene set size",
                y = "GSVA score") +
  ggplot2::theme_bw()

What do you notice about these distributions? How might you use this information to inform your interpretation of GSVA scores?

How can you use these scores?

If you did have groups information for your samples, you could test for pathway scores differences between groups. Here’s an example of that from the OpenPBTA project itself!

You can also visualize this matrix in a heatmap. Here’s a figure from the OpenPBTA project, where the middle panel is a heatmap of GSVA scores that were significantly different between histologies.

Code here.

Write results to file

gsva_results %>%
  as.data.frame() %>%
  tibble::rownames_to_column("pathway") %>%
  readr::write_tsv(file = gsva_results_file)

Session Info

sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8   
 [6] LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C        
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] GSVA_1.38.2    magrittr_2.0.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6                  mvtnorm_1.1-1               qusage_2.24.0               lattice_0.20-41            
 [5] tidyr_1.1.2                 fftw_1.0-6                  assertthat_0.2.1            digest_0.6.27              
 [9] R6_2.5.0                    GenomeInfoDb_1.26.2         stats4_4.0.3                RSQLite_2.2.3              
[13] evaluate_0.14               coda_0.19-4                 httr_1.4.2                  ggplot2_3.3.3              
[17] pillar_1.4.7                zlibbioc_1.36.0             rlang_0.4.10                rstudioapi_0.13            
[21] annotate_1.68.0             blob_1.2.1                  S4Vectors_0.28.1            Matrix_1.3-2               
[25] rmarkdown_2.6               labeling_0.4.2              BiocParallel_1.24.1         readr_1.4.0                
[29] stringr_1.4.0               RCurl_1.98-1.2              bit_4.0.4                   munsell_0.5.0              
[33] DelayedArray_0.16.2         compiler_4.0.3              xfun_0.20                   pkgconfig_2.0.3            
[37] BiocGenerics_0.36.0         htmltools_0.5.1.1           tidyselect_1.1.0            SummarizedExperiment_1.20.0
[41] GenomeInfoDbData_1.2.4      tibble_3.0.5                IRanges_2.24.1              matrixStats_0.57.0         
[45] XML_3.99-0.5                fansi_0.4.2                 crayon_1.3.4                dplyr_1.0.3                
[49] bitops_1.0-6                grid_4.0.3                  nlme_3.1-151                jsonlite_1.7.2             
[53] xtable_1.8-4                GSEABase_1.52.1             gtable_0.3.0                lifecycle_0.2.0            
[57] DBI_1.1.1                   scales_1.1.1                graph_1.68.0                cli_2.2.0                  
[61] estimability_1.3            stringi_1.5.3               farver_2.0.3                XVector_0.30.0             
[65] limma_3.46.0                ellipsis_0.3.1              generics_0.1.0              vctrs_0.3.6                
[69] tools_4.0.3                 bit64_4.0.5                 Biobase_2.50.0              glue_1.4.2                 
[73] purrr_0.3.4                 hms_1.0.0                   MatrixGenerics_1.2.0        emmeans_1.5.3              
[77] parallel_4.0.3              yaml_2.2.1                  AnnotationDbi_1.52.0        colorspace_2.0-0           
[81] GenomicRanges_1.42.0        memoise_1.1.0               knitr_1.30
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzIChHU1ZBKSIKb3V0cHV0OiAKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjAKLS0tCgpTbyBmYXIgZXZlcnkgcGF0aHdheSBhbmFseXNpcyBtZXRob2Qgd2UndmUgY292ZXJlZCByZWxpZXMgb24gc29tZSBpbmZvcm1hdGlvbiBhYm91dCBncm91cHMgb2Ygc2FtcGxlcyBpbiBvdXIgZGF0YS4KRm9yIG92ZXItcmVwcmVzZW50YXRpb24gYW5hbHlzaXMgKE9SQSksIHdlIGNyZWF0ZWQgZ2VuZSBzZXRzIGZyb20gdHdvIGRpZmZlcmVudCB0d28gZ3JvdXAgY29tcGFyaXNvbnMuCkluIHRoZSBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzIChHU0VBKSBleGFtcGxlLCB3ZSB1c2VkIHN0YXRpc3RpY3MgZnJvbSBhIGRpZmZlcmVudGlhbCBnZW5lIGV4cHJlc3Npb24gKERHRSkgYW5hbHlzaXMgd2hlcmUgd2UgY29tcGFyZWQgX01ZQ05fIGFtcGxpZmllZCBjZWxsIGxpbmVzIHRvIG5vbi1hbXBsaWZpZWQgY2VsbCBsaW5lczsgd2UgbmVlZGVkIHRoYXQgYW1wbGlmaWNhdGlvbiBzdGF0dXMgaW5mb3JtYXRpb24uCgpXaGF0IGlmIHdlJ3JlIGxlc3Mgc3VyZSBhYm91dCBncm91cHMgaW4gb3VyIGRhdGEgb3Igd2Ugd2FudCB0byBhbmFseXplIG91ciBkYXRhIGluIGEgbW9yZSB1bnN1cGVydmlzZWQgbWFubmVyPwoKSW4gdGhpcyBub3RlYm9vayB3ZSB3aWxsIGNvdmVyIGEgbWV0aG9kIGNhbGxlZCBHZW5lIFNldCBWYXJpYXRpb24gQW5hbHlzaXMgKEdTVkEpIChbSMOkbnplbG1hbm4gX2V0IGFsLl8gMjAxM10oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpKSB0aGF0IGFsbG93cyB1cyB0byBjYWxjdWxhdGUgZ2VuZSBzZXQgb3IgcGF0aHdheSBzY29yZXMgb24gYSBwZXItc2FtcGxlIGJhc2lzLgoKV2UgbGlrZSB0aGlzIHF1b3RlIGZyb20gdGhlIEdTVkEgcGFwZXIgKFtIw6RuemVsbWFubiBfZXQgYWwuXyAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpIHRvIHNldCB0aGUgc3RhZ2U6Cgo+IFdoaWxlIFtnZW5lIHNldCBlbnJpY2htZW50XSBtZXRob2RzIGFyZSBnZW5lcmFsbHkgcmVnYXJkZWQgYXMgZW5kIHBvaW50cyBvZiBhIGJpb2luZm9ybWF0aWMgYW5hbHlzaXMsIEdTVkEgY29uc3RpdHV0ZXMgYSBzdGFydGluZyBwb2ludCB0byBidWlsZCBwYXRod2F5LWNlbnRyaWMgbW9kZWxzIG9mIGJpb2xvZ3kuCgpSYXRoZXIgdGhhbiBjb250ZXh0dWFsaXppbmcgc29tZSByZXN1bHRzIHlvdSBfYWxyZWFkeSBoYXZlXyBmcm9tIGFub3RoZXIgYW5hbHlzaXMgbGlrZSBER0UsIEdTVkEgaXMgZGVzaWduZWQgdG8gcHJvdmlkZSBhbiBlc3RpbWF0ZSBvZiBwYXRod2F5IHZhcmlhdGlvbiBmb3IgZWFjaCBvZiB0aGUgc2FtcGxlcyBpbiBhbiBleHBlcmltZW50LgpOb3RlIHRoYXQgdGhlc2Ugc2NvcmVzIHdpbGwgZGVwZW5kIG9uIHRoZSBzYW1wbGVzIGluY2x1ZGVkIGluIHRoZSBkYXRhc2V0IHdoZW4geW91IHJ1biBHU1ZBOyBpZiB5b3UgYWRkZWQgbW9yZSBzYW1wbGVzIGFuZCByZXJhbiBHU1ZBLCB5b3Ugd291bGQgZXhwZWN0IHRoZSBzY29yZXMgdG8gY2hhbmdlLgoKIyMjIyBPdGhlciByZXNvdXJjZXMKCiogW01hbGhvdHJhLiBfRGVjb2RpbmcgR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzXy4gKDIwMTgpXShodHRwczovL3Rvd2FyZHNkYXRhc2NpZW5jZS5jb20vZGVjb2RpbmctZ2VuZS1zZXQtdmFyaWF0aW9uLWFuYWx5c2lzLTgxOTNhMGNmZGEzKQoKIyMgU2V0IHVwCgojIyMgTGlicmFyaWVzCgpgYGB7ciBsaWJyYXJpZXN9CiMgUGlwZXMKbGlicmFyeShtYWdyaXR0cikKIyBHZW5lIFNldCBWYXJpYXRpb24gQW5hbHlzaXMKbGlicmFyeShHU1ZBKQpgYGAKCiMjIyBEaXJlY3RvcmllcyBhbmQgZmlsZXMKCiMjIyMgRGlyZWN0b3JpZXMKCmBgYHtyIGRpcmVjdG9yaWVzfQojIFdlIGhhdmUgc29tZSBtZWR1bGxvYmxhc3RvbWEgZGF0YSBmcm9tIHRoZSBPcGVuUEJUQSBwcm9qZWN0IHRoYXQgd2UndmUgCiMgcHJlcGFyZWQgYWhlYWQgb2YgdGltZQppbnB1dF9kaXIgPC0gZmlsZS5wYXRoKCJkYXRhIiwgIm9wZW4tcGJ0YSIpCgojIENyZWF0ZSBhIGRpcmVjdG9yeSBzcGVjaWZpY2FsbHkgZm9yIHRoZSByZXN1bHRzIHVzaW5nIHRoaXMgZGF0YXNldApvdXRwdXRfZGlyIDwtIGZpbGUucGF0aCgicmVzdWx0cyIsICJvcGVuLXBidGEiKSAKaWYgKCFkaXIuZXhpc3RzKG91dHB1dF9kaXIpKSB7CiAgZGlyLmNyZWF0ZShvdXRwdXRfZGlyLCByZWN1cnNpdmUgPSBUUlVFKQp9CmBgYAoKIyMjIyBJbnB1dAoKV2UgaGF2ZSBWU1QgdHJhbnNmb3JtZWQgUk5BLXNlcSBkYXRhLCBhbm5vdGF0ZWQgd2l0aCBnZW5lIHN5bWJvbHMsIHRoYXQgaGFzIGJlZW4gY29sbGFwc2VkIHN1Y2ggdGhhdCB0aGVyZSBhcmUgbm8gZHVwbGljYXRlZCBnZW5lIGlkZW50aWZpZXJzIChzZWUgYHNldHVwYCkuCgpgYGB7ciBpbnB1dF9maWxlfQpybmFzZXFfZmlsZSA8LSBmaWxlLnBhdGgoaW5wdXRfZGlyLCAibWVkdWxsb2JsYXN0b21hX3ZzdF9jb2xsYXBzZWQudHN2IikKYGBgCgojIyMjIE91dHB1dAoKYGBge3Igb3V0cHV0X2ZpbGUsIGxpdmUgPSBUUlVFfQpnc3ZhX3Jlc3VsdHNfZmlsZSA8LSBmaWxlLnBhdGgob3V0cHV0X2RpciwgIm1lZHVsbG9ibGFzdG9tYV9nc3ZhX3Jlc3VsdHMudHN2IikKYGBgCgojIyBHZW5lIHNldHMKClRoZSBmdW5jdGlvbiB0aGF0IHdlIHdpbGwgdXNlIHRvIHJ1biBHU1ZBIHdhbnRzIHRoZSBnZW5lIHNldHMgdG8gYmUgaW4gYSBsaXN0LCByYXRoZXIgdGhhbiBhIHRpZHkgZGF0YSBmcmFtZSB0aGF0IHdlIHVzZWQgd2l0aCBgY2x1c3RlclByb2ZpbGVyYCAoYWx0aG91Z2ggaXQgZG9lcyBhY2NlcHQgb3RoZXIgZm9ybWF0cykuCgpXZSdyZSBnb2luZyB0byB0YWtlIHRoaXMgb3Bwb3J0dW5pdHkgdG8gaW50cm9kdWNlIGEgZGlmZmVyZW50IGZvcm1hdCB0aGF0IGdlbmUgc2V0cyBhcmUgb2Z0ZW4gZGlzdHJpYnV0ZWQgaW4gY2FsbGVkIFtHTVQgKEdlbmUgTWF0cml4IFRyYW5zcG9zZWQpXShodHRwczovL3NvZnR3YXJlLmJyb2FkaW5zdGl0dXRlLm9yZy9jYW5jZXIvc29mdHdhcmUvZ3NlYS93aWtpL2luZGV4LnBocC9EYXRhX2Zvcm1hdHMjR01UOl9HZW5lX01hdHJpeF9UcmFuc3Bvc2VkX2ZpbGVfZm9ybWF0Xy4yOC4yQS5nbXQuMjkpLgoKV2UncmUgZ29pbmcgdG8gcmVhZCBpbiB0aGUgSGFsbG1hcmsgY29sbGVjdGlvbiBmaWxlIGRpcmVjdGx5IGZyb20gW01TaWdEQl0oaHR0cHM6Ly93d3cuZ3NlYS1tc2lnZGIub3JnL2dzZWEvbXNpZ2RiL2luZGV4LmpzcCksIHJhdGhlciB0aGFuIHVzaW5nIGBtc2lnZGJyYCBsaWtlIHdlIGRpZCBpbiBlYXJsaWVyIG5vdGVib29rcy4KClRoZSBSTkEtc2VxIGRhdGEgdXNlcyBnZW5lIHN5bWJvbHMsIHNvIHdlIG5lZWQgZ2VuZSBzZXRzIHRoYXQgdXNlIGdlbmUgc3ltYm9scywgdG9vLgoKYGBge3IgZ210fQojIFIgY2FuIG9mdGVuIHJlYWQgaW4gZGF0YSBmcm9tIGEgVVJMCmhhbGxtYXJrc191cmwgPC0gImh0dHBzOi8vZGF0YS5icm9hZGluc3RpdHV0ZS5vcmcvZ3NlYS1tc2lnZGIvbXNpZ2RiL3JlbGVhc2UvNy4xL2guYWxsLnY3LjEuc3ltYm9scy5nbXQiCgojIFF1U0FHRSBpcyBhbm90aGVyIHBhdGh3YXkgYW5hbHlzaXMgbWV0aG9kLCB0aGUgcXVzYWdlIHBhY2thZ2UgaGFzIGEgZnVuY3Rpb24KIyBmb3IgcmVhZGluZyBHTVQgZmlsZXMgYW5kIHR1cm5pbmcgdGhlbSBpbnRvIGEgbGlzdApoYWxsbWFya3NfbGlzdCA8LSBxdXNhZ2U6OnJlYWQuZ210KGhhbGxtYXJrc191cmwpCmBgYAoKV2hhdCBkb2VzIHRoaXMgbGlzdCBsb29rIGxpa2U/CgpgYGB7ciBoZWFkX2hhbGxtYXJrLCBldmFsID0gRkFMU0V9CmhlYWQoaGFsbG1hcmtzX2xpc3QpCmBgYAoKIyMgUk5BLXNlcSBkYXRhCgpXZSBoYXZlIFZTVCB0cmFuc2Zvcm1lZCBSTkEtc2VxIGRhdGEsIHdoaWNoIGlzIG9uIGEgbG9nMi1saWtlIHNjYWxlLgpUaGVzZSBkYXRhIGFyZSBmcm9tIHRoZSBPcGVuIFBlZGlhdHJpYyBCcmFpbiBUdW1vciBBdGxhcyAoT3BlblBCVEEpCk9wZW5QQlRBIGlzIGEgY29sbGFib3JhdGl2ZSBwcm9qZWN0IG9yZ2FuaXplZCBieSB0aGUgQ0NETCBhbmQgdGhlIENlbnRlciBmb3IgRGF0YS1Ecml2ZW4gRGlzY292ZXJ5IGluIEJpb21lZGljaW5lIChEM2IpIGF0IHRoZSBDaGlsZHJlbidzIEhvc3BpdGFsIG9mIFBoaWxhZGVscGhpYSBjb25kdWN0ZWQgb3Blbmx5IG9uIEdpdEh1Yi4KCllvdSBjYW4gcmVhZCBtb3JlIGFib3V0IHRoZSBwcm9qZWN0IFtoZXJlXShodHRwczovL2dpdGh1Yi5jb20vYWxleHNsZW1vbmFkZS9vcGVucGJ0YS1hbmFseXNpcy8jb3BlbnBidGEtYW5hbHlzaXMpLgoKV2UncmUgb25seSB3b3JraW5nIHdpdGggdGhlIG1lZHVsbG9ibGFzdG9tYSBzYW1wbGVzIGluIHRoaXMgZXhhbXBsZS4KCmBgYHtyIHJlYWRfaW5fcm5hc2VxfQpybmFzZXFfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KHJuYXNlcV9maWxlKQpgYGAKCmBgYHtyIHBlZWtfcm5hc2VxLCBsaXZlID0gVFJVRX0KIyBXaGF0IGRvZXMgdGhlIFJOQS1zZXEgZGF0YSBmcmFtZSBsb29rIGxpa2U/CnJuYXNlcV9kZlsxOjUsIDE6NV0KYGBgCgpGb3IgR1NWQSwgd2UgbmVlZCBhIG1hdHJpeC4KCmBgYHtyIHJuYXNlcV9tYXQsIGxpdmUgPSBUUlVFfQpybmFzZXFfbWF0IDwtIHJuYXNlcV9kZiAlPiUKICB0aWJibGU6OmNvbHVtbl90b19yb3duYW1lcygiZ2VuZV9zeW1ib2wiKSAlPiUKICBhcy5tYXRyaXgoKQpgYGAKCipOb3RlOiBJZiB3ZSBoYWQgZHVwbGljYXRlIGdlbmUgc3ltYm9scyBoZXJlLCB3ZSBjb3VsZG4ndCBzZXQgdGhlbSBhcyByb3duYW1lcy4qCgojIyBHU1ZBCgohW10oZGlhZ3JhbXMvaGFuemVsbWFubl9maWcxLmpwZykKCioqRmlndXJlIDEgZnJvbSBbSMOkbnplbG1hbm4gX2V0IGFsLl8gKDIwMTMpXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykuKioKCllvdSBtYXkgbm90aWNlIHRoYXQgR1NWQSBoYXMgc29tZSBjb21tb25hbGl0aWVzIHdpdGggR1NFQS4KUmF0aGVyIHRoYW4gcmFua2luZyBnZW5lcyBiYXNlZCBvbiBzb21lIHN0YXRpc3RpYyBfd2VfIHNlbGVjdGVkIGFoZWFkIG9mIHRpbWUsIEdTVkEgZml0cyBhIG1vZGVsIGFuZCByYW5rcyBnZW5lcyBiYXNlZCBvbiB0aGVpciBleHByZXNzaW9uIGxldmVsIHJlbGF0aXZlIHRvIHRoZSBzYW1wbGUgZGlzdHJpYnV0aW9uLgpUaGlzIGlzIGEgd2F5IG9mIGFza2luZyBpZiBhIGdlbmUgX2lfIGlzIGhpZ2hseSBvciBsb3dseSBleHByZXNzZWQgaW4gYSBzYW1wbGUgX2pfIGluIHRoZSBjb250ZXh0IG9mIHRoaXMgZXhwZXJpbWVudCBhbmQgcmFua2luZyBhY2NvcmRpbmdseSAoW0jDpG56ZWxtYW5uIF9ldCBhbC5fIDIwMTNdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTg2LzE0NzEtMjEwNS0xNC03KSkuClRoZSBwYXRod2F5LWxldmVsIHNjb3JlIGNhbGN1bGF0ZWQgaXMgYSB3YXkgb2YgYXNraW5nIGhvdyBnZW5lcyBfd2l0aGluXyBhIGdlbmUgc2V0IHZhcnkgYXMgY29tcGFyZWQgdG8gZ2VuZXMgdGhhdCBhcmUgX291dHNpZGVfIG9mIHRoYXQgZ2VuZSBzZXQgKFtNYWxob3RyYS4gMjAxOF0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL2RlY29kaW5nLWdlbmUtc2V0LXZhcmlhdGlvbi1hbmFseXNpcy04MTkzYTBjZmRhMykpLiAKKFRoaXMgaXMgc29tZXRpbWVzIGNhbGxlZCBhIGNvbXBldGl0aXZlIHRlc3QgaW4gZ2VuZSBzZXQgZW5yaWNobWVudCBsaXRlcmF0dXJlLikKVGhlIGludHVpdGlvbiBoZXJlIGlzIHRoYXQgd2Ugd2lsbCBnZXQgcGF0aHdheS1sZXZlbCBzY29yZXMgZm9yIGVhY2ggc2FtcGxlIHRoYXQgaW5kaWNhdGUgaWYgZ2VuZXMgaW4gYSBwYXRod2F5IHZhcnkgY29uY29yZGFudGx5IGluIG9uZSBkaXJlY3Rpb24gKG92ZXJleHByZXNzZWQgb3IgdW5kZXJleHByZXNzZWQgcmVsYXRpdmUgdG8gdGhlIG92ZXJhbGwgcG9wdWxhdGlvbikgKFtIw6RuemVsbWFubiBfZXQgYWwuXyAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpLgoKVGhlIG91dHB1dCBpcyBhIGdlbmUgc2V0IGJ5IHNhbXBsZSBtYXRyaXggb2YgR1NWQSBzY29yZXMuCgojIyMgUGVyZm9ybSBHU1ZBCgpgYGB7ciBydW5fZ3N2YX0KZ3N2YV9yZXN1bHRzIDwtIGdzdmEocm5hc2VxX21hdCwgCiAgICAgICAgICAgICAgICAgICAgIGhhbGxtYXJrc19saXN0LAogICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAiZ3N2YSIsCiAgICAgICAgICAgICAgICAgICAgICMgQXBwcm9wcmlhdGUgZm9yIG91ciB0cmFuc2Zvcm1lZCBkYXRhIG9uIGxvZzItbGlrZSBzY2FsZQogICAgICAgICAgICAgICAgICAgICBrY2RmID0gIkdhdXNzaWFuIiwgIAogICAgICAgICAgICAgICAgICAgICAjIE1pbmltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICBtaW4uc3ogPSAxNSwgCiAgICAgICAgICAgICAgICAgICAgICMgTWF4aW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgIG1heC5zeiA9IDUwMCwKICAgICAgICAgICAgICAgICAgICAgIyBDb21wdXRlIEdhdXNzaWFuLWRpc3RyaWJ1dGVkIHNjb3JlcwogICAgICAgICAgICAgICAgICAgICBteC5kaWZmID0gVFJVRSkKYGBgCgoqKk5vdGU6IHRoZSBgZ3N2YSgpYCBkb2N1bWVudGF0aW9uIHNheXMgd2UgY2FuIHVzZSBga2NkZiA9ICJHYXVzc2lhbiJgIGlmIHdlIGhhZCBSTkEtc2VxIGxvZy1DUE1zLCBsb2ctUlBLTXMgb3IgbG9nLVRQTXMsIGJ1dCB3ZSB3b3VsZCB1c2UgYGtjZGYgPSAiUG9pc3NvbiJgIG9uIGludGVnZXIgY291bnRzLioqCgpgYGB7ciBnc3ZhX3BlZWt9CiMgTGV0J3MgZXhwbG9yZSB3aGF0IHRoZSBvdXRwdXQgb2YgZ3N2YSgpIGxvb2tzIGxpa2UKZ3N2YV9yZXN1bHRzWzE6NSwgMTo1XQpgYGAKCiMjIyBBIG5vdGUgb24gZ2VuZSBzZXQgc2l6ZQogCk9mdGVuIHRoZSBzY29yZXMgb2YgZ2VuZSBzZXQgZW5yaWNobWVudCBtZXRob2RzIGFyZSBub3QgY29tcGFyYWJsZSBiZXR3ZWVuIGdlbmUgc2V0cyBvZiBkaWZmZXJlbnQgc2l6ZXMuCkxldCdzIGRvIGFuIGV4cGVyaW1lbnQgdXNpbmcgcmFuZG9tbHkgZ2VuZXJhdGVkIGdlbmUgc2V0cyB0byBleHBsb3JlIHRoaXMgaWRlYSBhIGJpdCBtb3JlLgoKV2UgbmVlZCB0byBnZXQgYSBjb2xsZWN0aW9uIG9mIGFsbCBwb3NzaWJsZSBnZW5lcyB3ZSB3aWxsIHNhbXBsZSBmcm9tIHRvIGNyZWF0ZSByYW5kb20gZ2VuZSBzZXRzLgpCZWNhdXNlIHdlJ3JlIGRvaW5nIHNvbWUgcmFuZG9tIHNhbXBsaW5nLCB3ZSBuZWVkIHRvIHNldCBhIHNlZWQgZm9yIHRoaXMgdG8gYmUgcmVwcm9kdWNpYmxlLgoKYGBge3IgcmFuZG9tX3NldHVwfQojIFVzZSBhbGwgdGhlIGdlbmUgc3ltYm9scyBpbiB0aGUgZGF0YXNldCBhcyB0aGUgcG9vbCBvZiBwb3NzaWJsZSBnZW5lcwphbGxfZ2VuZXMgPC0gcm93bmFtZXMocm5hc2VxX21hdCkKCiMgU2V0IGEgc2VlZCBmb3IgcmVwcm9kdWNpYmlsaXR5CnNldC5zZWVkKDIwMjApCmBgYAoKT3VyIG1pbmltdW0gZ2VuZSBzZXQgc2l6ZSBlYXJsaWVyIHdhcyAxNSBnZW5lcyBhbmQgb3VyIG1heGltdW0gZ2VuZSBzZXQgc2l6ZSB3YXMgNTAwIGdlbmVzLiBXZSdsbCB1c2UgdGhlIHNhbWUgbWluaW11bSBhbmQgbWF4aW11bSB2YWx1ZXMgZm9yIG91ciByYW5kb20gZ2VuZSBzZXRzIGFuZCBzb21lIHZhbHVlcyBpbiBiZXR3ZWVuLgoKYGBge3IgZ2VuZV9zZXRfc2l6ZXN9CiMgTWFrZSBhIGxpc3Qgb2YgaW50ZWdlcnMgdGhhdCBpbmRpY2F0ZSB0aGUgcmFuZG9tIGdlbmUgc2V0IHNpemVzCmdlbmVfc2V0X3NpemUgPC0gbGlzdCgxNSwgMjUsIDUwLCAxMDAsIDI1MCwgNTAwKQpgYGAKCkZvciBlYWNoIGdlbmUgc2V0IHNpemUsIHdlIHdpbGwgZ2VuZXJhdGUgMTAwIHJhbmRvbSBnZW5lIHNldHMKCmBgYHtyIHJhbmRvbV9nZW5lX3NldHN9CiMgU2V0IG51bWJlciBvZiByZXBsaWNhdGVzCm5yZXBzIDwtICAxMDAKIyBHZW5lcmF0ZSAxMDAgcmFuZG9tIGdlbmUgc2V0cyBvZiBlYWNoIHNpemUKcmFuZG9tX2dlbmVfc2V0cyA8LSAKICBwdXJycjo6bWFwKAogICAgcmVwKGdlbmVfc2V0X3NpemUsIG5yZXBzKSwgICMgUmVwZWF0IGdlbmUgc2l6ZXMgc28gd2UgcnVuIGBucmVwc2AgdGltZXMgCiAgICAjIFNhbXBsZSB0aGUgdmVjdG9yIG9mIGFsbCBnZW5lcywgY2hvb3NpbmcgdGhlIG51bWJlciBvZiBpdGVtcyBzcGVjaWZpZWQKICAgICMgaW4gdGhlIGVsZW1lbnQgb2YgZ2VuZSBzZXQgc2l6ZQogICAgfiBiYXNlOjpzYW1wbGUoeCA9IGFsbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIHNpemUgPSAueCkKICApCmBgYAoKVGhlIEhhbGxtYXJrcyBsaXN0IHdlIHVzZWQgZWFybGllciBzdG9yZWQgdGhlIGdlbmUgc2V0IG5hbWVzIGFzIHRoZSBuYW1lIG9mIHRoZSBsaXN0LCBzbyBsZXQncyBhZGQgbmFtZXMgdG8gb3VyIHJhbmRvbSBnZW5lIHNldHMgdGhhdCBpbmRpY2F0ZSB3aGF0IHNpemUgdGhleSBhcmUgYW5kIHNvIGBnc3ZhKClgIGRvZXNuJ3QgZ2V0IHVwc2V0LgoKYGBge3IgbmFtZV9yYW5kb21fZ2VuZV9zZXRzfQojIFdlIHdpbGwgaW5jbHVkZSB0aGUgc2l6ZSBvZiB0aGUgZ2VuZSBzZXQgaW4gdGhlIGdlbmUgc2V0IG5hbWUgCiMgU3RhcnQgYnkgdGFraW5nIHRoZSBsZW5ndGggb2YgZWFjaCBwYXRod2F5IGFuZCBhcHBlbmRpbmcgInBhdGh3YXlfIiB0byB0aGF0CiMgbnVtYmVyCmxlbmd0aHNfdmVjdG9yIDwtIHB1cnJyOjptYXAocmFuZG9tX2dlbmVfc2V0cywgfiBsZW5ndGgoLngpKSAlPiUKICBwdXJycjo6bWFwKH4gcGFzdGUwKCJwYXRod2F5XyIsIC54KSkgJT4lCiAgIyBSZXR1cm4gYSB2ZWN0b3IKICBwdXJycjo6ZmxhdHRlbl9jaHIoKQoKIyBBZGQgdGhlIG5hbWVzIGluIGxlbmd0aHNfdmVjdG9yIHRvIHRoZSBsaXN0CnJhbmRvbV9nZW5lX3NldHMgPC0gcmFuZG9tX2dlbmVfc2V0cyAlPiUgCiAgIyBtYWtlLm5hbWVzKCkgYXBwZW5kcyBhICJ2ZXJzaW9uIiBpZiBzb21ldGhpbmcgaXMgbm90IHVuaXF1ZQogIHB1cnJyOjpzZXRfbmFtZXMobm0gPSBtYWtlLm5hbWVzKGxlbmd0aHNfdmVjdG9yLCB1bmlxdWUgPSBUUlVFKSkKYGBgCiAKUnVuIEdTVkEgb24gb3VyIGRhdGFzZXQgd2l0aCB0aGUgc2FtZSBwYXJhbWV0ZXJzIGFzIGJlZm9yZSwgYnV0IG5vdyB3aXRoIHJhbmRvbSBnZW5lIHNldHMuCgpgYGB7ciByYW5kb21fZ3N2YSwgbGl2ZSA9IFRSVUV9CnJhbmRvbV9nc3ZhX3Jlc3VsdHMgPC0gZ3N2YShybmFzZXFfbWF0LCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJhbmRvbV9nZW5lX3NldHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAiZ3N2YSIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIEFwcHJvcHJpYXRlIGZvciBvdXIgdHJhbnNmb3JtZWQgZGF0YSBvbiAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgbG9nMi1saWtlIHNjYWxlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBrY2RmID0gIkdhdXNzaWFuIiwgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBNaW5pbXVtIGdlbmUgc2V0IHNpemUKICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1pbi5zeiA9IDE1LCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgTWF4aW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBtYXguc3ogPSA1MDAsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIENvbXB1dGUgR2F1c3NpYW4tZGlzdHJpYnV0ZWQgc2NvcmVzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBteC5kaWZmID0gVFJVRSkgICAgICAgICAgICAgICAgICAgICAgICAgICAgCmBgYAoKTm93IGxldCdzIG1ha2UgYSBwbG90IHRvIGxvb2sgYXQgdGhlIGRpc3RyaWJ1dGlvbiBvZiBzY29yZXMgZnJvbSByYW5kb20gZ2VuZSBzZXRzLgpGaXJzdCB3ZSBuZWVkIHRvIGdldCB0aGlzIGRhdGEgaW4gYW4gYXBwcm9wcmlhdGUgZm9ybWF0IGZvciBgZ2dwbG90MmAuCgpgYGB7ciBsb25nZXJfcmFuZG9tX2dzdmF9CiMgVGhlIHJhbmRvbSByZXN1bHRzIGFyZSBhIG1hdHJpeApyYW5kb21fbG9uZ19kZiA8LSByYW5kb21fZ3N2YV9yZXN1bHRzICU+JQogIGRhdGEuZnJhbWUoKSAlPiUKICAjIEdlbmUgc2V0IG5hbWVzIGFyZSByb3duYW1lcwogIHRpYmJsZTo6cm93bmFtZXNfdG9fY29sdW1uKCJnZW5lX3NldCIpICU+JQogICMgR2V0IGludG8gbG9uZyBmb3JtYXQKICB0aWR5cjo6cGl2b3RfbG9uZ2VyKGNvbHMgPSAtZ2VuZV9zZXQsIAogICAgICAgICAgICAgICAgICAgICAgbmFtZXNfdG8gPSAiS2lkc19GaXJzdF9CaW9zcGVjaW1lbl9JRCIsCiAgICAgICAgICAgICAgICAgICAgICB2YWx1ZXNfdG8gPSAiZ3N2YV9zY29yZSIpICU+JQogICMgUmVtb3ZlIHRoZSAudmVyc2lvbiBhZGRlZCBieSBtYWtlLm5hbWVzKCkKICBkcGx5cjo6bXV0YXRlKGdlbmVfc2V0ID0gc3RyaW5ncjo6c3RyX3JlbW92ZShnZW5lX3NldCwgIlxcLi4qIikpICU+JQogICMgQWRkIGEgY29sdW1uIHRoYXQga2VlcHMgdHJhY2sgb2YgdGhlIGdlbmUgc2V0IHNpemUKICBkcGx5cjo6bXV0YXRlKGdlbmVfc2V0X3NpemUgPSBzdHJpbmdyOjp3b3JkKGdlbmVfc2V0LCAyLCBzZXAgPSAiXyIpKSAlPiUKICAjIFdlIHdhbnQgdG8gcGxvdCBzbWFsbGVzdCBuby4gZ2VuZXMgLT4gbGFyZ2VzdCBuby4gZ2VuZXMKICBkcGx5cjo6bXV0YXRlKGdlbmVfc2V0X3NpemUgPSBmYWN0b3IoZ2VuZV9zZXRfc2l6ZSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxldmVscyA9IGMoMTUsIDI1LCA1MCwgMTAwLCAyNTAsIDUwMCkpKQoKYGBgCgpMZXQncyBtYWtlIGEgdmlvbGluIHBsb3Qgc28gd2UgY2FuIGxvb2sgYXQgdGhlIGRpc3RyaWJ1dGlvbiBvZiBzY29yZXMgYnkgZ2VuZSBzZXQgc2l6ZS4KCmBgYHtyIHJhbmRvbV92aW9saW59CiMgVmlvbGluIHBsb3QgY29tcGFyaW5nIEdTVkEgc2NvcmVzIG9mIGRpZmZlcmVudCByYW5kb20gZ2VuZSBzZXQgc2l6ZXMKcmFuZG9tX2xvbmdfZGYgJT4lCiAgZ2dwbG90Mjo6Z2dwbG90KGdncGxvdDI6OmFlcyh4ID0gZ2VuZV9zZXRfc2l6ZSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5ID0gZ3N2YV9zY29yZSkpICsKICAjIE1ha2UgYSB2aW9saW4gcGxvdCB0aGF0J3MgYSBwcmV0dHkgYmx1ZSEKICBnZ3Bsb3QyOjpnZW9tX3Zpb2xpbihmaWxsID0gIiM5OUNDRkYiLCBhbHBoYSA9IDAuNSkgKwogICMgQWRkIGEgcG9pbnQgd2l0aCB0aGUgbWVhbiB2YWx1ZQogIGdncGxvdDI6OnN0YXRfc3VtbWFyeSgKICAgIGdlb20gPSAicG9pbnQiLAogICAgZnVuID0gIm1lYW4iLAogICAgIyBDaGFuZ2UgdGhlIGFlc3RoZXRpY3Mgb2YgdGhlIHBvaW50cwogICAgc2l6ZSA9IDMsCiAgICBjb2xvciA9ICIjMDA2NkNDIiwKICAgIHNoYXBlID0gMTgKICApICsKICAjIEZsaXAgdGhlIGF4ZXMKICBnZ3Bsb3QyOjpjb29yZF9mbGlwKCkgKwogIGdncGxvdDI6OmxhYnModGl0bGUgPSAiUmFuZG9tIEdlbmUgU2V0IEdTVkEgU2NvcmVzIiwKICAgICAgICAgICAgICAgIHggPSAiZ2VuZSBzZXQgc2l6ZSIsCiAgICAgICAgICAgICAgICB5ID0gIkdTVkEgc2NvcmUiKSArCiAgZ2dwbG90Mjo6dGhlbWVfYncoKQpgYGAKCldoYXQgZG8geW91IG5vdGljZSBhYm91dCB0aGVzZSBkaXN0cmlidXRpb25zPwpIb3cgbWlnaHQgeW91IHVzZSB0aGlzIGluZm9ybWF0aW9uIHRvIGluZm9ybSB5b3VyIGludGVycHJldGF0aW9uIG9mIEdTVkEgc2NvcmVzPwoKIyMjIEhvdyBjYW4geW91IHVzZSB0aGVzZSBzY29yZXM/CgpJZiB5b3UgZGlkIGhhdmUgZ3JvdXBzIGluZm9ybWF0aW9uIGZvciB5b3VyIHNhbXBsZXMsIHlvdSBjb3VsZCB0ZXN0IGZvciBwYXRod2F5IHNjb3JlcyBkaWZmZXJlbmNlcyBiZXR3ZWVuIGdyb3Vwcy4KSGVyZSdzIFthbiBleGFtcGxlXShodHRwczovL2h0bWxwcmV2aWV3LmdpdGh1Yi5pby8/aHR0cHM6Ly9naXRodWIuY29tL0FsZXhzTGVtb25hZGUvT3BlblBCVEEtYW5hbHlzaXMvYmxvYi85YjQ0YmYxYzE4NmIzMTI2YjE2ZGJlNWI4Nzc1NmIzZWFlM2ZlZWMyL2FuYWx5c2VzL2dlbmUtc2V0LWVucmljaG1lbnQtYW5hbHlzaXMvMDItbW9kZWwtZ3NlYS5uYi5odG1sKSBvZiB0aGF0IGZyb20gdGhlIE9wZW5QQlRBIHByb2plY3QgaXRzZWxmISAKCllvdSBjYW4gYWxzbyB2aXN1YWxpemUgdGhpcyBtYXRyaXggaW4gYSBoZWF0bWFwLgpIZXJlJ3MgYSBmaWd1cmUgZnJvbSB0aGUgT3BlblBCVEEgcHJvamVjdCwgd2hlcmUgdGhlIG1pZGRsZSBwYW5lbCBpcyBhIGhlYXRtYXAgb2YgR1NWQSBzY29yZXMgdGhhdCB3ZXJlIHNpZ25pZmljYW50bHkgZGlmZmVyZW50IGJldHdlZW4gaGlzdG9sb2dpZXMuCgohW10oaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL0FsZXhzTGVtb25hZGUvT3BlblBCVEEtYW5hbHlzaXMvOWI0NGJmMWMxODZiMzEyNmIxNmRiZTViODc3NTZiM2VhZTNmZWVjMi9maWd1cmVzL3BuZ3MvdHJhbnNjcmlwdG9taWMtb3ZlcnZpZXcucG5nKQoKWypDb2RlIGhlcmUuKl0oaHR0cHM6Ly9naXRodWIuY29tL0FsZXhzTGVtb25hZGUvT3BlblBCVEEtYW5hbHlzaXMvYmxvYi85YjQ0YmYxYzE4NmIzMTI2YjE2ZGJlNWI4Nzc1NmIzZWFlM2ZlZWMyL2ZpZ3VyZXMvc2NyaXB0cy90cmFuc2NyaXB0b21pYy1vdmVydmlldy5SI0wxMDYpCgojIyMgV3JpdGUgcmVzdWx0cyB0byBmaWxlCgpgYGB7ciB3cml0ZV9nc3ZhX3Jlc3VsdHN9CmdzdmFfcmVzdWx0cyAlPiUKICBhcy5kYXRhLmZyYW1lKCkgJT4lCiAgdGliYmxlOjpyb3duYW1lc190b19jb2x1bW4oInBhdGh3YXkiKSAlPiUKICByZWFkcjo6d3JpdGVfdHN2KGZpbGUgPSBnc3ZhX3Jlc3VsdHNfZmlsZSkKYGBgCgojIyBTZXNzaW9uIEluZm8KCmBgYHtyIHNlc3Npb25faW5mb30Kc2Vzc2lvbkluZm8oKQpgYGAK